Q6_K - Block Interleaving Implementation for x86 SIMD (AVX512/AVX2)#19706
Open
Manogna-Sree wants to merge 3 commits intoggml-org:masterfrom
Open
Q6_K - Block Interleaving Implementation for x86 SIMD (AVX512/AVX2)#19706Manogna-Sree wants to merge 3 commits intoggml-org:masterfrom
Manogna-Sree wants to merge 3 commits intoggml-org:masterfrom
Conversation
github-actions
bot
added
the
ggml
Manogna-Sree
force-pushed
the
Q6_K_blockinterleaving_implementation
branch
from
d2d844c to
b886cf9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make sure to read the contributing guidelines before submitting a PR
PR #15275 which was previously submitted by us, includes repacking and block interleaving for the Q6K nodes. However, with the latest master, we observed inaccuracies related to the usage of repacked scales present in the master branch.
This PR fixes the inaccuracy issues and contains block interleaving approach for Q6_K quantization for x64/x86 SIMD Architecture
Initial gains were observed with prompt processing with the above changes compared to the tested Q6_K model
The GEMM function was implemented for AVX512/AVX2 and GEMV functions are implemented for the AVX2 architecture
GCC Version = 12.3
The PR was tested in AMD Granite Ridge 9600X which supports the following flags by default :
system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
Additionally, the PR was tested for execution with clang on both Linux and windows
The perplexity results with llama2 7B are tabulated as follows: